Verification of Markov Decision Processes with Risk-Sensitive Measures
نویسندگان
چکیده
We develop a method for computing policies in Markov decision processes with risk-sensitive measures subject to temporal logic constraints. Specifically, we use a particular risk-sensitive measure from cumulative prospect theory, which has been previously adopted in psychology and economics. The nonlinear transformation of the probabilities and utility functions yields a nonlinear programming problem, which makes computation of optimal policies typically challenging. We show that this nonlinear weighting function can be accurately approximated by the difference of two convex functions. This observation enables efficient policy computation using convexconcave programming. We demonstrate the effectiveness of the approach on several scenarios.
منابع مشابه
Risk-Sensitive Markov Control Processes
We introduce a unified framework to incorporate risk in Markov decision processes (MDPs), via prospect maps, which generalize the idea of coherent/convex risk measures in mathematical finance. Most of the existing risk-sensitive approaches in various literature concerning with decision-making problems are contained in the framework as special instances. Within the framework, we solve the optima...
متن کاملRisk-Sensitive and Average Optimality in Markov Decision Processes
Abstract. This contribution is devoted to the risk-sensitive optimality criteria in finite state Markov Decision Processes. At first, we rederive necessary and sufficient conditions for average optimality of (classical) risk-neutral unichain models. This approach is then extended to the risk-sensitive case, i.e., when expectation of the stream of one-stage costs (or rewards) generated by a Mark...
متن کاملIterated risk measures for risk-sensitive Markov decision processes with discounted cost
We demonstrate a limitation of discounted expected utility, a standard approach for representing the preference to risk when future cost is discounted. Specifically, we provide an example of the preference of a decision maker that appears to be rational but cannot be represented with any discounted expected utility. A straightforward modification to discounted expected utility leads to inconsis...
متن کاملAnalysis of a risk-sensitive control problem for hidden Markov chains
In this paper the risk-sensitive control of parially observed Markov decision processes is considered. The replacement problem is analyzed in this context, and the structure of risk sensitive optimal controllers is given.
متن کاملA Short Note on Combining Multiple Policies in Risk-Sensitive Exponential Average Reward Markov Decision Processes
This short note presents a method of combining multiple policies in a given policy set such that the resulting policy improves all policies in the set for risk-sensitive exponential average reward Markov decision processes (MDPs), extending the work of Howard and Matheson for the singleton policy set case. Some applications of the method in solving risk-sensitive MDPs are also discussed.
متن کامل